<html>
<head><meta charset="utf-8"><title>working with TokenStreams in rustc · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html">working with TokenStreams in rustc</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="217663871"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217663871" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Xavier Denis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217663871">(Nov 23 2020 at 18:18)</a>:</h4>
<p>hey, I know that this would be bending the rustc apis quite a ways, but is there anyway to turn a <code>rustc_ast</code> <code>TokenStream</code> into a <code>proc_macro</code> one? I've been staring at the implementation of <code>proc_macro_server</code> for a while and I can't figure out where the types actually change. </p>
<p>For context, I'm working on a rustc_driver and I'd like to run a <code>syn</code> macro on a <code>TokenStream</code> inside this driver</p>



<a name="217664480"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217664480" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217664480">(Nov 23 2020 at 18:24)</a>:</h4>
<p><span class="user-mention" data-user-id="312719">@Xavier Denis</span> I think it happens at <a href="https://github.com/rust-lang/rust/blob/40cf72108edb9b8633a9d284b238988309204494/compiler/rustc_expand/src/proc_macro.rs#L84">https://github.com/rust-lang/rust/blob/40cf72108edb9b8633a9d284b238988309204494/compiler/rustc_expand/src/proc_macro.rs#L84</a></p>



<a name="217664867"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217664867" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Xavier Denis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217664867">(Nov 23 2020 at 18:27)</a>:</h4>
<p>I might be confused but that seems to produce a <code>rustc_ast::TokenStream</code> for part of a rust ast</p>



<a name="217664934"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217664934" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Xavier Denis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217664934">(Nov 23 2020 at 18:28)</a>:</h4>
<p>I'm looking for the step afterwards where we get a <code>proc_macro::TokenStream</code></p>



<a name="217664980"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217664980" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Xavier Denis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217664980">(Nov 23 2020 at 18:28)</a>:</h4>
<p>and from what I can tell it happens inside <code>client.run</code> but I can't actually figure out where the 'meat' of it is</p>



<a name="217665077"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217665077" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Xavier Denis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217665077">(Nov 23 2020 at 18:29)</a>:</h4>
<p>do we just <code>transmute</code> the tokenstream betwene representatons? (by doing FFI)?</p>



<a name="217665690"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217665690" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217665690">(Nov 23 2020 at 18:35)</a>:</h4>
<p>On the server side any <code>TokenStream</code> type can be used. You just have to implement certain traits on your context (in this case <code>Rustc</code>: <a href="https://github.com/rust-lang/rust/blob/40cf72108edb9b8633a9d284b238988309204494/compiler/rustc_expand/src/proc_macro_server.rs#L353">https://github.com/rust-lang/rust/blob/40cf72108edb9b8633a9d284b238988309204494/compiler/rustc_expand/src/proc_macro_server.rs#L353</a>) for <code>proc_macro::server</code> to manipulate it. The <code>proc_macro::bridge::client::Client</code> then takes this server side <code>TokenStream</code> type and stores it somewhere. The client side then gets a token that will be stored in the client <code>TokenStream</code> that can then be used to ask the server for information about and manipulate the token stream.</p>



<a name="217665818"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217665818" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217665818">(Nov 23 2020 at 18:36)</a>:</h4>
<p>Here is the <code>TokenStream</code> handle type defined: <a href="https://github.com/rust-lang/rust/blob/40cf72108edb9b8633a9d284b238988309204494/library/proc_macro/src/bridge/client.rs#L161">https://github.com/rust-lang/rust/blob/40cf72108edb9b8633a9d284b238988309204494/library/proc_macro/src/bridge/client.rs#L161</a></p>



<a name="217665896"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217665896" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217665896">(Nov 23 2020 at 18:37)</a>:</h4>
<p>It may be easier to use <code>proc_macro2</code>, which can also be used outside proc-macro's.</p>



<a name="217666693"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217666693" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Xavier Denis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217666693">(Nov 23 2020 at 18:44)</a>:</h4>
<p>Hmm I see so the client/proc_macro token stream is fundamentally tied to the existence of a server</p>



<a name="217666737"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217666737" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Xavier Denis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217666737">(Nov 23 2020 at 18:44)</a>:</h4>
<p>Yea proc_macro2 would solve this but I’d need to turn a rustc_ast tokenstream into that one</p>



<a name="217666782"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/working%20with%20TokenStreams%20in%20rustc/near/217666782" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Xavier Denis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/working.20with.20TokenStreams.20in.20rustc.html#217666782">(Nov 23 2020 at 18:45)</a>:</h4>
<p>Maybe I’ll just keep my hacky code for now (turn everything into a string, parse that)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>